How to Use and Misuse Image Assessment Algorithms

Natural images are meaningful to humans and used for a variety of tasks. The specific task to be performed dictates the assessment of an image’s suitability to the task. This work considers image assessment (IA) as a task-specific evaluation and examines the performance of a variety of objective IA algorithms for three different assessment tasks: fidelity assessment, quality assessment, and utility assessment. Fidelity assessment refers to an evaluation of processed images with respect to the visibility of the distortions. Quality assessment refers to the evaluation of processed images with regard to the tolerance of visible distortions. Utility assessment refers to an evaluation of processed images based on their usefulness as a surrogate for the reference image. Objective IA algorithms have been developed that suit either the fidelity assessment or quality assessment tasks, but none have been considered for the utility assessment task. This work compares and contrasts these three assessment tasks. We discuss how the training database of subjective ratings used to validate the performance of an IA algorithm influences 1) the task associated with the algorithm, 2) the suitability of the algorithm to other tasks, and 3) the limitations of the IA algorithm. Present IA algorithms are shown to be ill-suited for the utility assessment task.